Active Grounding of Visual Situations

We address a key problem for computer vision: retrieving images that are instances of visual situations. Visual situations are concepts such as “a boxing match”, “a birthday party”, “walking the dog”, “a crowd waiting for a bus,” “a handshake”, or “a game of ping-pong,” whose instantiations in images are linked more by their common spatial and semantic structure than by low-level visual similarity. While computer vision has made remarkable progress on recognizing individual objects in images, the problem of visual situation recognition is much more difficult for many reasons, including the vast variability of possible instances of a given situation, as well as the combinatorics of evaluating possible pairwise or multiple-object relationships. In this paper we describe a novel architecture we have developed for visual situation retrieval. Given a situation description, our architecture—called Situate—learns models capturing the visual features of expected objects as well as probabilistic spatial models capturing the expected spatial configuration of relationships among objects. Given a new image, Situate uses these models in an attempt to ground (i.e., to create a bounding box representing) each expected component of the situation in the image via an active search procedure. Situate uses the resulting grounding to compute a score indicating the degree to which the new image is judged to contain an instance of the situation. Such scores can be used to rank images in a collection as part of a retrieval system. In the preliminary study described here, we demonstrate the promise of this system—and the importance of active grounding—by comparing Situate’s retrieval and grounding performance on one example situation category with that of two baseline methods, as well as with a related image-retrieval system based on “scene graphs”.